Non-parametric Bayesian Dictionary Learning with Landmark-Dependent Hierarchical Beta Process

There has been significant recent interest in dictionary learning and sparse coding, with applications in denoising, interpolation, feature extraction, and classification [1]–[3]. Increasingly it has been recognized that these models may be improved by imposing additional prior information, beyond sparseness. For example, a locality constraint has been used successfully in the context of feature learning and image classification [4]. Structured sparsity has been used for compressive sensing [5]. Other examples include hierarchical tree-based dictionary learning [6], submodular dictionary selection [7], and exploitation of self-similarity in images [8]. We propose a landmark-dependent hierarchical beta process to address dictionary learning for data that are endowed with an associated covariate. We explore the idea that data nearby in the covariate space (e.g., nearby in terms of temporal, spatial or cosine distances) are likely to share similar sparseness properties, and hence we employ “landmarks” to guide the usage probabilities of dictionary atoms. We address covariate-dependent dictionary learning from a Bayesian perspective. Compared to many optimization-based approaches, which assume the noise is Gaussian and its variance is known [1], our model has the advantage of automatically inferring an appropriate dictionary size and the underlying noise statistics. Our model is connected to recent research devoted to removing the exchangeability assumption of the IBP [9], [10] and BP [11]. The exchangeability assumption ignores relational information provided by covariates. A dependent IBP (dIBP) model has been introduced recently, with a hierarchical Gaussian process (GP) used to account for covariate dependence [12]. In the proposed model, rather than imposing relational information via a parametric covariance matrix, as in GP, we do so by employing a kernel-based construction. We introduce “landmarks” in the covariate space, whose positions are learned, defining local regions where the dictionary usages are likely to be similar. The normalized kernels are localized via learned “landmarks,” establishing links between data and landmark-dependent sparseness properties. The proposed model is related to the kernel stick breaking process (KSBP) [13] and Bayesian density regression (BDR) [14], although it is distinct from both. For example, the original KSBP construction focused primarily on covariate-dependent mixture models, and here we extend such ideas to a sparse factor analysis (SFA) model for covariate-dependent dictionary learning and sparse coding; we also